NAME¶

PCPIntro - introduction to the Performance Co-Pilot (PCP)

INTRODUCTION¶

The Performance Co-Pilot (PCP) is a toolkit designed for monitoring and managing system-level performance. These services are distributed and scalable to accommodate the most complex system configurations and performance problems.

PCP supports many different platforms, including (but not limited to) Linux, MacOSX, Solaris and Windows. From a high-level PCP can be considered to contain two classes of software utility:

PCP Collectors

These are the parts of PCP that collect and extract performance data from various sources, e.g. the operating system kernel.

PCP Monitors

These are the parts of PCP that display data collected from hosts (or archives) that have the PCP Collector installed. Many monitor tools are available as part of the core PCP release, while other (typically graphical) monitoring tools are available separately in the PCP GUI package.

This manual entry describes the high-level features and options common to most PCP utilities available on all platforms.

OVERVIEW¶

The PCP architecture is distributed in the sense that any PCP tool may be executing remotely. On the host (or hosts) being monitored, each domain of performance metrics, whether the kernel, a service layer, a database management system, a web server, an application, etc. requires a Performance Metrics Domain Agent (PMDA) which is responsible for collecting performance measurements from that domain. All PMDAs are controlled by the Performance Metrics Collector Daemon (pmcd(1)) on the same host.

Client applications (the monitoring tools) connect to pmcd(1), which acts as a router for requests, by forwarding requests to the appropriate PMDA and returning the responses to the clients. Clients may also access performance data from a PCP archive (created using pmlogger(1)) for retrospective analysis.

COMMON COMMAND LINE ARGUMENTS¶

There is a set of common command line arguments that are used consistently by most PCP tools.

-a archive

Performance metric information is retrospectively retrieved from the Performance Co-Pilot (PCP) archive, previously generated by pmlogger(1).See pcp-archive(5)forformatdocumentation. The -a and -h options are mutually exclusive.

-a archive[,archive,...]

An alternate form of -a for applications that are able to handle multiple archives.

-h hostname

Unless directed to another host by the -h option, or to an archive by the -a option, the source of performance metrics will be the Performance Metrics Collector Daemon (PMCD) on the local host. Refer to the PMCD HOST SPECIFICATION section later for further details on the many options available when forming the hostname specification, as well as a detailed description of the default local host connection. The -a and -h options are mutually exclusive.

-s samples

The argument samples defines the number of samples to be retrieved and reported. If samples is 0 or -s is not specified, the application will sample and report continuously (in real time mode) or until the end of the PCP archive (in archive mode).

-z

Change the reporting timezone to the local timezone at the host that is the source of the performance metrics, as identified via either the -h or -a options.

-Z timezone

By default, applications report the time of day according to the local timezone on the system where the application is executed. The -Z option changes the timezone to timezone in the format of the environment variable TZ as described in environ(7).

INTERVAL SPECIFICATION AND ALIGNMENT¶

Most PCP tools operate with periodic sampling or reporting, and the -t and -A options may be used to control the duration of the sample interval and the alignment of the sample times.

-t interval

number[units]

-A align

TIME WINDOW SPECIFICATION¶

Many PCP tools are designed to operate in some time window of interest, e.g. to define a termination time for real-time monitoring or to define a start and end time within a PCP archive log.

In the absence of the -O and -A options to specify an initial sample time origin and time alignment (see above), the PCP application will retrieve the first sample at the start of the time window.

The following options may be used to specify a time window of interest.

-S starttime

-T endtime

-O origin

The datetime argument for the -O, -S and -T options consists of:
date time zone day relative
A date can be one of: YY-MM-DD, MM/DD/YY, DD Month YYYY, or Month DD YYYY. A time can be one of: HH:MM:SS, HH:MM. HH:MM can use either the 12 hour (via an am or pm suffix) or 24 hour convention. A day of the week can be a spelled out day of the week, optionally preceded by an ordinal number such as second tuesday. A zone is a time zone value as specified by the tzselect(1) command. A relative time can be a time unit that is: preceded by a cardinal number such as 1 year or 2 months, preceded by one of the time words this or last, or succeeded by the time word ago. A relative time can also be one of the time words: yesterday, today, tomorrow, now. Examples of datetime strings are: 1996-03-04 13:07:47 EST Mon, 1996-03-05 14:07:47 EST -1hour, Mon Mar 4 13:07:47 1996, Mar 4 1996, Mar 4, Mar, 13:07:50 or 13:08.

For any missing low order fields, the default value of 0 is assumed for hours, minutes and seconds, 1 for day of the month and Jan for months. Hence, the following are equivalent: -S '@ Mar 1996' and -S '@ Mar 1 00:00:00 1996'.

If any high order fields are missing, they are filled in by starting with the year, month and day from the current time (real-time mode) or the time at the beginning of the PCP archive log (archive mode) and advancing the time until it matches the fields that are specified. So, for example if the time window starts by default at ``Mon Mar 4 13:07:47 1996'', then -S @13:10 corresponds to 13:10:00 on Mon Mar 4, 1996, while -S @10:00 corresponds to 10:00:00 on Tue Mar 5, 1996 (note this is the following day).

For greater precision than afforded by datetime(3), the seconds component may be a floating point number.

PERFORMANCE METRICS - NAMES AND IDENTIFIERS¶

The number of performance metric names supported by PCP on most platforms ranges from many hundreds to several thousand. The PCP libraries and applications use an internal identification scheme that unambiguously associates a single integer with each known performance metric. This integer is known as the Performance Metric Identifier, or PMID. Although not a requirement, PMIDs tend to have global consistency across all systems, so a particular performance metric usually has the same PMID.

For all users and most applications, direct use of the PMIDs would be inappropriate (e.g. this would limit the range of accessible metrics, make the code hard to maintain, force the user interface to be particularly baroque, etc.). Hence a Performance Metrics Name Space (PMNS) is used to provide external names and a hierarchic classification for performance metrics. A PMNS is represented as a tree, with each node having a label, a pointer to either a PMID (for leaf nodes) or a set of descendent nodes in the PMNS (for non-leaf nodes).

A node label must begin with an alphabetic character, followed by zero or more characters drawn from the alphabetics, the digits and character `_´ (underscore). For alphabetic characters in a node label, upper and lower case are distinguished.

By convention, the name of a performance metric is constructed by concatenation of the node labels on a path through the PMNS from the root node to a leaf node, with a ``.'' as a separator. The root node in the PMNS is unlabeled, so all names begin with the label associated with one of the descendent nodes below the root node of the PMNS, e.g. kernel.percpu.syscall. Typically (although this is not a requirement) there would be at most one name for each PMID in a PMNS. For example kernel.all.cpu.idle and disk.dev.read are the unique names for two distinct performance metrics, each with a unique PMID.

Groups of related PMIDs may be named by naming a non-leaf node in the PMNS tree, e.g. disk.

The default local PMNS used by pmcd is located at $PCP_VAR_DIR/pmns/root however the environment variable PMNS_DEFAULT may be set to the full pathname of a different PMNS which will then be used as the default local PMNS.

Most applications do not use the local PMNS directly, but rather import parts of the PMNS as required from the same place that performance metrics are fetched, i.e. from pmcd(1) for live monitoring or from a PCP archive for retrospective monitoring.

To explore the PMNS use pminfo(1), or if the PCP GUI package is installed the New Chart and Metric Search windows within pmchart(1).

PERFORMANCE METRIC SPECIFICATIONS¶

In configuration files and (to a lesser extent) command line options, metric specifications adhere to the following syntax rules.

If the source of performance metrics is real-time from pmcd(1) then the accepted syntax is
host:metric[instance1,instance2,...]

If the source of performance metrics is a PCP archive log then the accepted syntax is
archive/metric[instance1,instance2,...]

The host:, archive/ and [instance1,instance2,...] components are all optional.

The , delimiter in the list of instance names may be replaced by white space.

Special characters in instance names may be escaped by surrounding the name in double quotes or preceding the character with a backslash.

White space is ignored everywhere except within a quoted instance name.

An empty instance is silently ignored, and in particular ``[]'' is the same as no instance, while ``[one,,,two]'' is parsed as specifying just the two instances ``one'' and ``two''.

As a special case, if the host is the single character ``@'' then this refers to a PM_CONTEXT_LOCAL source, see pmNewContext(3).

SECURE PMCD CONNECTIONS¶

Since PCP version 3.6.11, a monitor can explicitly request a secure connection to a collector host running pmcd(1) or pmproxy(1) using the PM_CTXFLAG_SECURE context flag. If the PCP Collector host supports this feature - refer to the pmcd.feature.secure metric for confirmation of this - a TLS/SSL (Transport Layer Security or Secure Sockets Layer) connection can be established which uses public key cryptography and related techniques. These features aim to prevent eavesdropping and data tampering from a malicious third party, as well as providing server-side authentication (confident identification of a server by a client) which can be used to guard against man-in-the-middle attacks.

A secure pmcd connection requires use of certificate-based authentication. The security features offered by pmcd and pmproxy are implemented using the Network Security Services (NSS) APIs and utilities. The NSS certutil tool can be used to create certificates suitable for establishing trust between PCP monitor and collector hosts.

A complete description is beyond the scope of this document, refer to the PCP ENVIRONMENT, FILES and SEE ALSO sections for detailed information. This includes links to tutorials on the steps involved in setting up the available security features.

PMCD HOST SPECIFICATION¶

In the absence of an explicit host name specification, most tools will default to the local host in live update mode. In PCP releases since 3.8.4 onward, this results in an efficient local protocol being selected - typically a Unix domain socket. If this option is used (which can also be explicitly requested via the unix: host specification described below), it is important to note that all connections will be automatically authenticated. In other words, the credentials of the user invoking a client tool will automatically be made available to pmcd(1) and all of its PMDAs, on the users behalf, such that results can be customized to the privilege levels of individual users.

Names of remote hosts running the pmcd(1) daemon can of course also be provided to request a remote host be used. The most basic form of pmcd host specification is a simple host name, possibly including the domain name if necessary. However, this can be extended in a number of ways to further refine attributes of the connection made to pmcd.

The pmcd port number and also optional pmproxy(1) hostname and its port number, can be given as part of the host specification, since PCP version 3.0. These supersede (and override) the old-style PMCD_PORT, PMPROXY_HOST and PMPROXY_PORT environment variables.

The following are valid hostname specifications that specify connections to pmcd on host nas1.servers.com with/without a list of ports, with/without a pmproxy(1) connection through a firewall, and with IPv6 and IPv4 addresses as shown.

$ pcp -h nas1.servers.com:44321,4321@firewall.servers.com:44322
$ pcp -h nas1.servers.com:44321@firewall.servers.com:44322
$ pcp -h nas1.servers.com:44321@firewall.servers.com
$ pcp -h nas1.servers.com@firewall.servers.com
$ pcp -h nas1.servers.com:44321
$ pcp -h [fe80::2ad2:44ff:fe88:e4f1%p2p1]
$ pcp -h 192.168.0.103

In addition, ``connection attributes'' can also be specified. These include username, password (can be given interactively and may depend on the authentication mechanism employed), whether to target a specific running container, whether to use secure (encrypted) or native (naked) protocol, and so on. The previous examples all default to native protocol, and use no authentication. This can be altered, as in the following examples.

$ pcp -h pcps://app2.servers.com?container=cae8e6edc0d5
$ pcp -h pcps://nas1.servers.com:44321?username=tanya&method=gssapi
$ pcp -h pcps://nas2.servers.com@firewalls.r.us?method=plain
$ pcp -h pcp://nas3.servers.com
$ pcp -h 192.168.0.103?container=cae8e6edc0d5,method=digest-md5
$ pcp -h unix:
$ pcp -h local:

The choice of authentication method, and other resulting parameters like username, optionally password, etc, depends on the SASL2 configuration used by each (remote) pmcd. Tutorials are available specifying various aspects of configuring the authentication module(s) used, these fine details are outside the scope of this document.

In all situations, host names can be used interchangeably with IPv4 or IPv6 addressing (directly), as shown above. In the case of an IPv6 address, the full address must be enclosed by square brackets and the scope (interface) must also be specified.

The final local: example above is now the default for most tools. This connection is an automatically authenticated local host connection on all platforms that support Unix domain sockets. No password is required and authentication is automatic. This is also the most efficient (lowest overhead) communication channel available.

The difference between unix: and local: is that the former is a strict Unix domain socket specification (connection fails if it cannot connect that way), whereas the latter has a more forgiving fallback to using localhost (i.e. a regular Inet socket connection is used when Unix domain socket connections are unavailable).

ENVIRONMENT¶

In addition to the PCP run-time environment and configuration variables described in the PCP ENVIRONMENT section below, the following environment variables apply to all installations.

PCP_CONSOLE

When set, this changes the default console from /dev/tty (on Unix) or CON: (on Windows) to be the specified console. The special value of none can be used to indicate no console is available for use. This is used in places where console-based tools need to interact with the user, and in particular is used when authentication is being performed.

PCP_DEBUG

When set, this variable provides an alternate way to initialize the value used to control diagnostic and debug output (the default value is 0 to suppress this output). The value for $PCP_DEBUG should be a decimal integer, see pmdbg(1) for a description of the supported values.

PCP_DERIVED_CONFIG

When set, this variable defines a colon separated list of files and/or directories (the syntax is the same as for the $PATH variable for sh(1)). The components are expanded into a list of files as follows: if a component of $PCP_DERIVED_CONFIG is a file, then that file is added to the list, else if a component is a directory then recursive descent is used to enumerate all files below that directory and these are added to the list.

PCP_SECURE_SOCKETS

When set, this variable forces any monitor tool connections to be established using the certificate-based secure sockets feature. If the connections cannot be established securely, they will fail.

PCP_SECURE_DB_METHOD

With secure socket connections, the certificate and key database is stored using the sql: method by default. Use PCP_SECURE_DB_METHOD to override the default, most usually setting the value to the empty string (for the older database methods).

PCP_STDERR

Many PCP tools support the environment variable PCP_STDERR, which can be used to control where error messages are sent. When unset, the default behavior is that ``usage'' messages and option parsing errors are reported on standard error, other messages after initial startup are sent to the default destination for the tool, i.e. standard error for ASCII tools, or a dialog for GUI tools.

PMCD_CONNECT_TIMEOUT

When attempting to connect to a remote pmcd(1) on a machine that is booting, the connection attempt could potentially block for a long time until the remote machine finishes its initialization. Most PCP applications and some of the PCP library routines will abort and return an error if the connection has not been established after some specified interval has elapsed. The default interval is 5 seconds. This may be modified by setting PMCD_CONNECT_TIMEOUT in the environment to a real number of seconds for the desired timeout. This is most useful in cases where the remote host is at the end of a slow network, requiring longer latencies to establish the connection correctly.

PMCD_RECONNECT_TIMEOUT

When a monitor or client application loses a connection to a pmcd(1), the connection may be re-established by calling a service routine in the PCP library. However, attempts to reconnect are controlled by a back-off strategy to avoid flooding the network with reconnection requests. By default, the back-off delays are 5, 10, 20, 40 and 80 seconds for consecutive reconnection requests from a client (the last delay will be repeated for any further attempts after the fifth). Setting the environment variable PMCD_RECONNECT_TIMEOUT to a comma separated list of positive integers will re-define the back-off delays, e.g. setting PMCD_RECONNECT_TIMEOUT to ``1,2'' will back-off for 1 second, then attempt another connection request every 2 seconds thereafter.

PMCD_REQUEST_TIMEOUT

For monitor or client applications connected to pmcd(1), there is a possibility of the application "hanging" on a request for performance metrics or metadata or help text. These delays may become severe if the system running pmcd crashes, or the network connection is lost. By setting the environment variable PMCD_REQUEST_TIMEOUT to a number of seconds, requests to pmcd will timeout after this number of seconds. The default behavior is to be willing to wait 10 seconds for a response from every pmcd for all applications.

PMCD_WAIT_TIMEOUT

When pmcd(1) is started from $PCP_RC_DIR/pcp then the primary instance of pmlogger(1) will be started if the configuration flag pmlogger is chkconfig(8) enabled and pmcd is running and accepting connections.

PMNS_DEFAULT

If set, then interpreted as the full pathname to be used as the default local PMNS for pmLoadNameSpace(3). Otherwise, the default local PMNS is located at $PCP_VAR_DIR/pcp/pmns/root for base PCP installations.

PCP_COUNTER_WRAP

Many of the performance metrics exported from PCP agents have the semantics of counter meaning they are expected to be monotonically increasing. Under some circumstances, one value of these metrics may smaller than the previously fetched value. This can happen when a counter of finite precision overflows, or when the PCP agent has been reset or restarted, or when the PCP agent is exporting values from some underlying instrumentation that is subject to some asynchronous discontinuity.

The environment variable PCP_COUNTER_WRAP may be set to indicate that all such cases of a decreasing ``counter'' should be treated as a counter overflow, and hence the values are assumed to have wrapped once in the interval between consecutive samples. This ``wrapping'' behavior was the default in earlier PCP versions, but by default has been disabled in PCP release from version 1.3 on.

PMDA_PATH

The PMDA_PATH environment variable may be used to modify the search path used by pmcd(1) and pmNewContext(3) (for PM_CONTEXT_LOCAL contexts) when searching for a daemon or DSO PMDA. The syntax follows that for PATH in sh(1), i.e. a colon separated list of directories, and the default search path is ``/var/pcp/lib:/usr/pcp/lib'', (or ``/var/lib/pcp/lib'' on Linux, depending on the value of the $PCP_VAR_DIR environment variable).

PMCD_PORT

The TPC/IP port(s) used by pmcd(1) to create the socket for incoming connections and requests, was historically 4321 and more recently the officially registered port 44321; in the current release, both port numbers are used by default as a transitional arrangement. This may be over-ridden by setting PMCD_PORT to a different port number, or a comma-separated list of port numbers. If a non-default port is used when pmcd is started, then every monitoring application connecting to that pmcd must also have PMCD_PORT set in their environment before attempting a connection.

The following environment variables are relevant to installations in which pmlogger(1), the PCP archive logger, is used.

PMLOGGER_PORT

The environment variable PMLOGGER_PORT may be used to change the base TCP/IP port number used by pmlogger(1) to create the socket to which pmlc(1) instances will try and connect. The default base port number is 4330. When used, PMLOGGER_PORT should be set in the environment before pmlogger is executed.

PMLOGGER_REQUEST_TIMEOUT

When pmlc(1) connects to pmlogger(1), there is a remote possibility of pmlc "hanging" on a request for information as a consequence of a failure of the network or pmlogger. By setting the environment variable PMLOGGER_REQUEST_TIMEOUT to a number of seconds, requests to pmlogger will timeout after this number of seconds. The default behavior is to be willing to wait forever for a response from each request to a pmlogger. When used, PMLOGGER_REQUEST_TIMEOUT should be set in the environment before pmlc is executed.

If you have the PCP product installed, then the following environment variables are relevant to the Performance Metrics Domain Agents (PMDAs).

PMDA_LOCAL_PROC

Use this variable has been deprecated and it is now ignored. If the ``proc'' PMDA is configured as a DSO for use with pmcd(1) on the local host then all of the ``proc'' metrics will be available to applications using a PM_CONTEXT_LOCAL context.

PMDA_LOCAL_SAMPLE

Use this variable has been deprecated and it is now ignored. If the ``sample'' PMDA is configured as a DSO for use with pmcd(1) on the local host then all of the ``sample'' metrics will be available to applications using a PM_CONTEXT_LOCAL context.

PMIECONF_PATH

If set, pmieconf(1) will form its pmieconf(5) specification (set of parameterized pmie(1) rules) using all valid pmieconf files found below each subdirectory in this colon-separated list of subdirectories. If not set, the default is $PCP_VAR_DIR/config/pmieconf.

FILES¶

/etc/pcp.conf

Configuration file for the PCP runtime environment, see pcp.conf(5).

/etc/pki/nssdb

Optionally contains a Network Security Services database with a "PCP Collector" certificate providing trusted identification for the collector host.

$HOME/.pcp

User-specific directories containing configuration files for customisation of the various monitor tools, such as pmchart(1).

$HOME/.pki/nssdb

A shared Network Security Services (NSS) database directory containing per-user certificates identifying known valid remote pmcd collector hosts. The NSS certutil tool is one of several that can be used to maintain this database.

$PCP_RC_DIR/pcp

Script for starting and stopping pmcd(1).

$PCP_PMCDCONF_PATH

Control file for pmcd(1).

$PCP_PMCDOPTIONS_PATH

Command line options passed to pmcd(1) when it is started from $PCP_RC_DIR/pcp. All the command line option lines should start with a hyphen as the first character. This file can also contain environment variable settings of the form "VARIABLE=value".

$PCP_BINADM_DIR

Location of PCP utilities for collecting and maintaining PCP archives, PMDA help text, PMNS files etc.

$PCP_PMDAS_DIR

Parent directory of the installation directory for Dynamic Shared Object (DSO) PMDAs.

$PCP_RUN_DIR/pmcd.pid

If pmcd is running, this file contains an ascii decimal representation of its process ID.

$PCP_LOG_DIR/pmcd

Default location of log files for pmcd(1), current directory for running PMDAs. Archives generated by pmlogger(1) are generally below $PCP_LOG_DIR/pmlogger.

$PCP_LOG_DIR/pmcd/pmcd.log

Diagnostic and status log for the current running pmcd(1) process. The first place to look when there are problems associated with pmcd.

$PCP_LOG_DIR/pmcd/pmcd.log.prev

Diagnostic and status log for the previous pmcd(1) instance.

$PCP_LOG_DIR/NOTICES

Log of pmcd(1) and PMDA starts, stops, additions and removals.

$PCP_VAR_DIR/config

Contains directories of configuration files for several PCP tools.

$PCP_SYSCONF_DIR/pmcd/rc.local

Local script for controlling PCP boot, shutdown and restart actions.

$PCP_VAR_DIR/pmns

Directory containing the set of PMNS files for all installed PMDAs.

$PCP_VAR_DIR/pmns/root

The ASCII pmns(5) exported by pmcd(1) by default. This PMNS is be the super set of all other PMNS files installed in $PCP_VAR_DIR/pmns.

In addition, if the PCP product is installed the following files and directories are relevant.

$PCP_LOG_DIR/NOTICES

In addition to the pmcd(1) and PMDA activity, may be used to log alarms and notices from pmie(1) via pmpost(1).

$PCP_PMLOGGERCONTROL_PATH

Control file for pmlogger(1) instances launched from $PCP_RC_DIR/pcp and/or managed by pmlogger_check(1) and pmlogger_daily(1) as part of a production PCP archive collection setup.

PCP ENVIRONMENT¶

Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

SEE ALSO¶

pmcd(1), pmie(1), pmie_daily(1), pminfo(1), pmlc(1), pmlogger(1), pmlogger_daily(1), pmrep(1), pmstat(1), pmval(1), pcp(1), pcp.conf(5), pcp.env(5), pmns(5) and chkconfig(8).

If the PCP GUI package is installed, then the following entries are also relevant:
pmchart(1), pmtime(1), and pmdumptext(1).

If the secure sockets extensions have been enabled, then the following references are also relevant:
http://www.pcp.io/documentation.html
http://www.mozilla.org/projects/security/pki/nss/#documentation
http://www.mozilla.org/projects/security/pki/nss/tools/certutil.html

Also refer to the books Performance Co-Pilot User's and Administrator's Guide and Performance Co-Pilot Programmer's Guide which can be found at http://www.pcp.io/